Improving Adatree with soft splitting rules

We extend the framework of Adaboost so that it builds a smoothed decision tree rather than a neural network. The proposed method, “Adatree 2”, is derived from the assumption of a probabilistic observation model. It avoids the problem of over-fitting that appears in other tree-growing methods by reweighing the training examples, rather than splitting the training dataset at each node. It differs from Adaboost by steering the input data towards weak classifiers that are “tuned” to the conditional probability determined by the output of previously evaluated classifiers. As a consequence, Adatree 2 enjoys a lower computation cost than Adaboost. After defining this method, we present some early experimental results and identify some issues left open for future research. 1 Boosting and decision trees Boosting [1, 2] refers to methods to build weighed combinations of “weak classifiers”. At each training round, a weak classifier is added to the existing combination. These methods reduce the classification error on a training dataset at a geometric rate and have been shown to have good properties [3]. They are known to often not overfit and generalize well even when training is continued way beyond perfect classification of the training dataset. However, the computation cost of a boosted classifier increases linearly with the number of training rounds, so that, in computation-sensitive situations, some adaptations [4, 5, 6, 7, 8] are done to benefit from the good properties of boosting at a reasonable computation cost. It has been noted [9] that decision tree-growing can also be viewed as a boosting mechanism, in the sense that the training error can be made to decrease geometrically. However, decision trees are notorious for over-fitting [10, Ch. 3.7] and tree “pruning”, “shrinking” [11, cited by [12]] and “smoothing” [11, 13] are techniques developed to address this problem. Even then, over-fitting remains and decision trees have been combined by boosting to improve the generalization ability. The over-fitting of decision trees can be attributed to the fact that each node is determined by only the examples that reach it during training. Consequently, deeper nodes are built from dwindling training sets that are less and less representative of the general population. This is a very slightly modified version of an article submitted to NIPS 2004. Table 1: Training and classification processes, of Adaboost, decision trees and Adatree 2. Classification Training Adaboost • All classifiers are evaluated. • Output is linear combination of all classifiers. • Each classifier is trained with all examples, appropriately weighed. Adatree 2 • Only weak classifiers on path of input are evaluated. • Output is linear combination of evaluated classifiers. • Each classifier is trained with all examples, appropriately weighed. Decision Tree • Only weak classifiers on path of input are evaluated. • Output is that of last classifier or (smoothed trees) a linear combination of evaluated classifiers. • Each classifier is trained with the fraction of examples that reaches it. The proposed method, “Adatree 2” differs in that respect by diminishing the training weight of examples rather than discarding them altogether. This re-weighing scheme is derived by assuming a probabilistic observation model that will be introduced in Section 3. Given the observation model, we derive Adatree 2 by minimizing the expected classification error on the training dataset, in a similar manner to the minimization of the classification error in Adaboost. In Section 4, we will show that different assumptions on the observation model yield variants of Adatree 2 that are equivalent to Adaboost or to our previous method [14], which we shall call “Adatree 1” in the sequel. Table 1 compares the training and classification stages of Adaboost, decision trees and Adatree 2. This section also states the main problem that we leave open for future study, namely the choice of an observation model. Section 5 presents some preliminary experimental results that suggest that Adatree 2 has a practical potential : its generalization ability appear less good than that of Adaboost while improving over Adatree 1. Also, Adatree 2 improves over Adaboost in terms of computation cost. Finally, some concluding notes are given in Section 6. 2 Classifier model We consider classifiers which are weighed sums of weak classifiers hs () :